The field of security information/event management (SIM or SEM) is generally concerned with 1) collecting data from networks and networked devices that reflects network activity and/or operation of the devices and 2) analyzing the data to enhance security. For example, the data can be analyzed to identify an attack on the network or a networked device and determine which user or machine is responsible. If the attack is ongoing, a countermeasure can be performed to thwart the attack or mitigate the damage caused by the attack. The data that is collected usually originates in a message (such as an event, alert, or alarm) or an entry in a log file, which is generated by a networked device. Networked devices include firewalls, intrusion detection systems, and servers.
Each message or log file entry (“event”) is stored for future use. Security systems may also generate events, such as correlation events and audit events. Together with messages and log file entries, these and other events are also stored on disk. In an average customer deployment, one thousand events per second may be generated. This amounts to 100 million events per day or three billion events per month The analysis and processing of such a vast amount of data can incur significant load on the security system, causing delays in reporting results.